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(57) Abstract: A method and system are provided to report the 
findings of an expert's analysis of image data. The method and 
system are based on a reporting system that forms the basis of 
an image management system that can efficiently and systemati- 
cally generate image reports, facilitate data entry into searchable 
databases for data mining, and expedite billing and collections 
for the expert's services. The expert identifies a significant find- 
ing on an image and attaches a location: description code to the 
location of that finding in order to create a significant finding 
and an entry into a database. Further descriptions of that find- 
ing, such as dimensional measurements, may be automatically 
appended to the finding as secondary attributes. After the evalu- 
ation, the system sorts the findings in the database and presents 
the findings by prioritized categories. The expert edits and ap- 
proves a multimedia report which may be delivered by electronic 
means to an end-user. 
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Field Of The Invention 

The present invention relates generally to an image reporting method 
and system and more particularly to a method and computer-implemented 
procedure for creating electronic, multimedia reports based on a new structured 
reporting paradigm. 

Background Of The Invention 

Image reporting as currently practiced suffers from a lack of 
standardization, consistency, accountability, and efficiency. A root cause of 
these problems is the manner in which reports are generated, beginning with the 
lack of a standardized report format, particularly in the medical field of 
radiology. 

Radiologists generally review images of a body structure and dictate 
narrative descriptions of their image findings followed by summary statements. 
Clerical workers then transcribe the dictated statements and either print 
applicable reports or enter such information into a computerized radiology 
information system (RIS) and/or hospital information system (HIS). As a result, 
the content and format of radiology reports often vary greatly depending on the 
-1- 
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differing preferences and styles of individual radiologists. This inconsistency 
among the radiologists' reporting styles often hinders the correlation of the 
reported findings with the actual images by the recipients of the reports. 
Variability in the reporting styles also impedes on-going monitoring of specific 
findings from different examinations on the same patient, a task that is critical 
for patient care and time-consuming for radiologists. Further, traditional 
radiology reporting practices do not support data mining, a powerful tool which 
is useful in clinical trials, epidemiology studies, and outcomes analyses. 

In addition, conventional reporting practices often provide no 
mechanism to allow the radiologist to account for the effective communication 
of critical report information to the recipient. Frequently, radiologists 
mistakenly assume that when a report is approved and sent to a referring medical 
professional, their responsibility ends. To the contrary, radiologists are often 
held accountable for ensuring that proper action is taken on significant findings 
and are held liable for malpractice when proper action is not taken. 

Clinicians are the typical end-users of reports from radiologists. A 
major complaint of such clinicians against radiologists and their reporting 
practices involves point of service. This problem is illustrated by the following 
scenario: a patient receives emergency room x-rays for an injury during the 
night; a radiologist interprets the x-ray images the next morning; and, following 
transcription, a report is finally delivered to the emergency room physician, but 
typically only after the patient has been treated and released. Clinicians are now 
demanding that radiologists issue reports immediately after an imaging study has 
been performed. 

-2- 
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Hence, there is a pressing need to provide a reporting system which 
offers a standardized report format, enables consistency among reports, accounts 
for effective information flow, provides for quick turnaround of information to 
the end-user, provides for on-going tracking of previous findings, and supports 
data mining for public health statistics. In addition, these needs extend beyond 
the field of radiology, and include other medical fields such as pathology, 
histology, cardiology, dermatology, as well as other image analysis fields such 
as satellite imagery and photography. 



Summary of the Invention 

The present invention relates to a new reporting method and system 
for reporting the findings of an expert's analysis of image data and, more 
specifically, to a computer system and computer-implemented method for 
reporting an expert's findings relative to an analysis of image data. The method 
and system are based on a new structured reporting paradigm. The paradigm 
forms the basis of a radiology practice management system that can efficiently 
and systematically generate radiology reports, facilitate data entry into 
searchable databases, track findings, support clinical trials and outcomes 
analyses, and expedite hospital billing and collections. One fundamental aspect 
of this paradigm is that a user, e.g. an expert-radiologist, identifies a significant 
feature on an image and attaches a location: description code. The 
location: description code can describe what or who is present in the image, 
when the image was taken, where the image was taken, and how the image was 
taken. For example, in the case of consumer digital photography, the user can 
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attach a location: description code to a digital photograph in order to create a 
finding that indicates "Aunt Minnie: Vacationing at the beach", or in the case of 
radiology can attach an anatomicahpathological code to the location of an 
image feature to create a diagnostic finding. In the case of medical imaging, the 
anatomical-.pathological code includes the anatomical location followed by a 
pathological description. 

Optionally, further attributes of that finding, such as follow-up 
treatment or diagnosis recommendations, a priority descriptor, dimensional 
measurements (e.g., length, area, and volume), audio descriptions, 3D rendered 
snapshots, etc., may be automatically appended to the diagnostic finding as 
secondary attributes of the diagnostic finding. All of this information is 
automatically captured in an intuitive workflow scheme transparent to the 
expert, and stored in a database. The expert may continue to identify additional 
diagnostically significant features and create diagnostic findings in any order. 

At the end of the expert's evaluation of the image(s), the system sorts 
the diagnostic findings by selected or predetermined categories. In a medical 
field, these predetermined categories may be anatomical categories. The 
diagnostic findings are further prioritized by the severity of the diagnosis, e.g., 
the priority descriptor, in order to alert the report recipient, such as a clinician. 
In addition, the system may alert the expert and display a summary of significant 
prior findings, including trend data, relating to the present image analysis. The 
expert can edit and approve a multimedia report, which may be delivered to an 
Internet server for immediate access, sent to a database, sent by automated voice, 
fax, e-mail, or wireless personal digital assistant (PDA) (e.g. Palm™ handheld) 
-4- 
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to the clinician, or any combination thereof. The radiologist can sign the report 
by electronic or voice signature. The final report presentation may be further 
customized to satisfy the needs of the clinician. 

The reporting system of the present invention is applicable to several 
other image-based fields including, without limitation, pathology, histology, 
cardiology, dermatology, satellite imagery, and photography. 

Brief Description Of The Drawing s 

The foregoing summary and the following detailed description of the 
preferred embodiments of the present invention will be best understood when 
read in conjunction with the appended drawings, in which: 

Figure 1 illustrates a flowchart representing a general method in 
accordance with the present invention for creating an image report; 

Figure 2 illustrates a block diagram of a computer system used in the 
computer-implemented method of the present invention; 

Figure 3 illustrates a flowchart representing the steps of the process for 
creating an image report; 

Figure 4 illustrates a flowchart representing steps of operation of the 
method of the present invention; 

Figures 5A and 5B illustrate the steps of annotating findings; 

Figures 6 illustrates the user-interface of the present invention in which 
Figure 6A shows a 2D viewer and Figure 6B shows a 3D viewer; 

Figures 7A-7C illustrate a selected report of the present invention; 

Figures 8A-8D illustrate anatomy and pathology hot menus; 
-5- 
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Figures 9A-9C illustrate the display of significant prior findings; and 
Figure 10 illustrates a report showing the tracking of a disease. 



Detailed Description Of The Invention 

A method and system are provided for generating and communicating 
reports containing an expert's analysis of image data as generally depicted in 
Figs.l and 2. In addition, a computer-implemented method and a computer 
system function to create a database of the expert's findings from which a report 
is generated and from which data mining and other analyses may be conducted. 
The database can be a computer searchable database and maybe a relational 
computer database. 

The method and system of the present invention are applicable to any 
field which relates to a user's analysis of images. In particular, however, the 
method and system of the present invention are well-suited to image analysis 
found in medical applications. As such, the method and system of the present 
invention are illustrated in the accompanying figures and description in terms of 
the medical field of radiology. 

The method and system are particularly well-suited to the analysis of 
digital images. However, the method and system may also be adapted for use 
with analog images such as conventional x-ray films and conventional 
photographs. For example, the system can utilize a digital camera to load a 
digital representation of an analog image into computer memory for further 
processing. 
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The computerized reporting system 10 is designed to interface with 
existing information systems such as a Hospital Information System (HIS) 36, a 
Radiology Information System (RIS) 34, and a Picture Archiving and 
Communication System (PACS) 32, and to conform to certain standards 
including but not limited to the Digital Imaging and Communications in 
Medicine (DICOM) standard, DICOM Structured Reporting (SR) standard, or 
the Radiological Society of North America's Integrating the Healthcare 
Enterprise (THE) initiative. The reporting system 10 includes an examination 
image storage 22, a computer console 24, a computer 26, display(s) 28, and an 
input device 27. For illustration purposes the input device 27 is a three-button 
computer mouse, where the left and middle-mouse buttons (LMB, MMB) are 
used, for example, to manipulate image data, and the right-mouse button (RMB) 
is used, for example, to identify a new diagnostically significant feature and to 
start a database recording process. Other known input devices including LCD 
graphics tablets and touch-screens may be used as well as other custom devices. 
For example, an intelligent view box and digital camera device can be used with 
conventional x-rays. 

Bidirectional communication between the reporting system 10 and the 
information systems 32, 34, 36 allows the reporting system 10 to retrieve data 
from the information systems 32, 34, 36 and to update information in these 
systems to provide the desired report generated by the reporting system 10. For 
example, the reporting system 10 may download image data corresponding to 
radiological examinations of patients from the PACS 32, and additional 
information including but not limited to patient demographics, billing 
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information, laboratory data, and pathology reports from the HIS 36. The PACS 
32 stores information according to existing standards such as (DICOM). The 
data from the PACS 32 is stored in the examination image storage 22 where it 
can be accessed via the computer console 24 and computer 26 for display on the 
display 28. Alternatively, the reporting system 10 can directly access the PACS 
images without the need for an intermediate storage device, such as image 
storage 22. Additionally, the reporting system 10 maybe linked to 
communication systems such as the Internet, e-mail systems, fax, telephone, 
wireless communications systems such as pagers and cellphones, wireless 
PDA's and other communication systems. 

Referring now to Figs. 1 and 3 which illustrate the general method and 
detailed process steps of the present invention, respectively, preparation of a 
report begins with the loading of patient data, including but not limited to 
billing, demographics, laboratory data, pathology reports, and image data, step 
100. A file loader from computer 26 searches the examination storage 22 for 
examination data files available for analysis and displays the corresponding 
names of patients in a user-interface at step 300. Upon selection of a particular 
patient by the radiologist, the file loader displays all of the associated unread 
examination files for that patient. The radiologist selects a particular 
examination file, and the file loader loads the corresponding data into computer 
memory at step 302. The file loader searches through the image data in the 
selected examination and organizes the images by DICOM series (or any 
additional subdivisions),at step 304, prior to display in 2D, and optional 3D, 
viewers. 
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The file loader also displays the Current Procedural Terminology (CPT) 
and International Classification of Diseases (ICD) codes assigned to the selected 
examination and determines if they correlate at steps 102 and 306. (CPT codes 
describe the type of radiologic examination, and ICD codes indicate the reasons 
for performing a particular examination.) Proper matching of these codes are 
essential for reimbursement by health care insurers. The file loader compares 
the ICD and CPT codes and displays an alert if the codes are incompatible. The 
radiologist verifies the codes and enters any necessary changes. Correct 
assignment of these codes at the beginning of an examination is effected by the 
reporting system 10 to intelligently guide the presentation of diagnostic code 
menus during the annotation process described below. Prior to the review 
process, an anatomical-location menu and a pathology-description menu are 
initialized using the CPT codes at step 308. For example, if the CPT code 
describes that the examination encompasses a computed tomography (CT) scan 
of a patient's abdomen, then the anatomical-location menu may present specific 
elements such as "liver, spleen, pancreas, kidneys, etc" and not necessarily 
anatomical locations corresponding to head and extremity body structures. Of 
course, the system allows a user to access a "master index" (or dictionary) of all 
anatomical and pathological codes. In addition, the CPT codes guide default 
settings for window/level (W/L) settings, 3D opacity map settings, and report 
organization/layout. Likewise, a series menu is initialized to list all of the 
DICOM series available in the selected examination file at step 308. In addition, 
the file loader retrieves existing "new paradigm" reports, i.e., those created using 
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the present invention, from the patient's previous examinations and makes them 
available for review during the current study. 

One problem that can be encountered in the use of the anatomical- 
location and pathology-description menus is that the radiologist can be 
presented with so many menu choices that an inordinate amount of time is 
required to make a menu selection. Such a large number of choices may 
unnecessarily increase the cost of creating a report and may create user fatigue 
and increased risk for a repetitive stress injury. Accordingly, it would be 
desirable to limit the choices presented to the radiologist. The system of the 
present invention addresses such problems through the use of "hot menus." 

A "hot menu" may utilize information from the CPT codes, ICD codes 
existing reports which have been loaded at step 308, and/or user trends 
determined by data mining analysis of the user's prior reports to tailor the menus, 
such as the anatomical-location menu and pathology-description menu. For 
example, the data contained in the current report and/or in any previous reports 
can be used to reorder the presentation of menu choices, so that the selections 
used most recently and/or most frequently are presented earlier in the menus. 
For example, if the most recently selected anatomy was "spleen", then "spleen" 
would appear at the top of the anatomical-location menu in a hot menu section, 
as shown in Figs. 8A - 8C. A separator, such as a row of asterisks or a line, 
could be placed below menu choices that are moved to a higher level of the 
menu to differentiate these choices from the choices located at their standard 
positions within the menu. Similarly, the most recently or most frequently 
selected pathology-descriptions, as determined from pre-existing reports, may 
-10- 
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be presented as hot menu selections at the top of the pathology-description 
menu, as shown in Fig. 8D. As an alternative to a simultaneous presentation of 
hot menu selections with ordinary menu items, the menu may show only the 
recent and/or frequent selections followed by a click-able menu entry which 
expands to show the standard options when the user clicks on that entry. 

One rationale for this approach is that an organ is usually evaluated in its 
entirety at any given time. If the right kidney contains multiple cysts, then the 
user may want to click on several cysts in the right kidney in rapid succession. 
In such a case it would be very efficient to see kidney appear at the top of the 
anatomy menu and the term cyst appears at the top of the pathology menu. 
Likewise, it would be very efficient to see the term "right" appear as the default 
in the characteristic field. 

In addition, the anatomical and pathological menu presentations may be 
modified by removing irrelevant choices from the anatomical-location menu or 
pathology-description menu. Data present in the loaded reports can be analyzed 
for certain information, such as the patient's sex, and the anatomical-location 
and/or pathology-description menus can be modified appropriately. For 
example, the anatomical-location menu associated with genitalia can be tailored 
to include only genitalia choices appropriate to the sex of the patient. As a 
further example, if a particular diagnostic procedure, as indicated by the CPT 
codes, is incapable of detecting a certain diagnosis, then menu choices 
associated with that diagnosis can be removed. Likewise, if information in the 
image data indicates that the image data relates to the brain, then menu choices 
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associated with the neck, chest, abdomen, pelvis, and extremities need not be 
included. 

After initialization of the menus, the first available image from the sorted 
images is displayed in a user-interface by a 2D viewer 610 as shown in Fig. 6 A 
from which the radiologist may begin analysis of the first image, at steps 104 
and 310. Alternately, the radiologist is free to select a different DICOM series 
for evaluation from the series menu. For example, a CT or MRI examination 
often consists of multiple series, whereas a chest x-ray may contain only one 
series. Two or more series may also be displayed simultaneously (e.g., supine 
and prone series of a virtual colonoscopy study). When a single window display 
is employed, a "previous" button allows the radiologist to toggle between the 
two most recently viewed series. A window/level menu, W/L, is available as 
part of the user-interface which lists preset window and level settings (i.e., 
grayscale settings) for the 2D viewer. The preset settings can be specified in an 
options menu or modified using window/level sliders. 

The step of displaying and rendering images, step 310, includes altering 
the display of the images in response to commands from the radiologist. For 
example, the radiologist can pan through a number of images in the 2D viewer 
as the mouse is moved and the LMB is pressed, provided that more than one 
image is contained in the series. Similarly, the 2D viewer can translate (i.e., 
move) the image up/down and sideways when the mouse is moved and the 
MMB is pressed. The 2D viewer can also zoom the image display when the 
mouse is moved and the LMB and MMB are pressed simultaneously. An 
ovei-view button is provided in the user-interface to re-center the image in case 
-12- 
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the scene is moved out of sight. However, re-centering may be unnecessary if 
the ability to move or zoom an image is restricted. 

A 3D viewer is also provided in the user-interface, as shown in Fig. 6B 
to render images in step 3 10. A 2D/3D toggle button is also included in the 
user-interface to allow the radiologist to toggle between the 2D and 3D viewers 
at step 310. In the 3D viewer, the mouse operations are similar to those of the 
2D viewer except that pressing the LMB while moving the mouse causes the 3D 
rendered scene to rotate in space. The LMB can also be used to control a "fly- 
through" mode as used in virtual endoscopy as disclosed in U.S. Patent 
5,782,762. 

The 3D viewer incorporates techniques including render around a point 
and hybrid rendering (i.e., combined volume rendering, surface rendering, and 
multiplanar [MPR] display). These techniques are the subjects of previous U.S. 
Patents 5,782,762 and 5,920,319, the disclosures of which are incorporated 
herein by reference. When surface rendering and MPR are utilized, 
identification of new diagnostically significant features, discussed below, within 
the 3D environment works in the same fashion, with a RMB click. When the 
3D viewer is activated after a diagnostic finding has been created, the 
volume-rendered image, e.g., a cube of CT data, (or surface-rendered or MPR 
image(s) ) is centered around the coordinates of the diagnostic finding. 

A render-box-size menu is also provided in the user-interface to control 
the size of the volume (i.e., cube of digital data) rendered in the 3D viewer. 
When changing the volume size, the 3D display automatically adjusts the scene 
to fill the screen. An opacity-map menu, Op, in the 3D viewer permits the 
-13- 
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radiologist to control the level of transparency and grayscale/color scale of a 3D 
volume rendering. In addition, the opacity map function (or transparency) and 
grayscale/color map can be set by the system in response to the selection of an 
anatomicalpathological code by the radiologist, as described below. 
Furthermore, the system provides functions to manipulate the 3D object/scene 
including but not limited to zoom, cutting plane, and opaque cutting plane 
features. 

As a further aspect of the display step 310, an orientation button is 
provided in the user-interface to allow the radiologist to properly set the 
orientation of the image data prior to 3D rendering. For example, it is assumed 
that the 2D first image in a CT series is the most superior (i.e., highest) image, 
the patient's left is on the radiologist's right, and the patient's anterior surface is 
facing up. If the series needs to be reoriented, the radiologist can pan through 
the collection of images to locate the most superior image (or close to it). The 
radiologist then toggles the orientation button, at which time the 2D viewer goes 
into an orientation mode. The radiologist freely rotates the image plane by 
pressing the 1MB and moving the mouse until the proper anterior/posterior and 
left/right orientation is achieved. Finally, the radiologist toggles the orientation 
button again to set the proper orientation. The 3D viewer then automatically 
adjusts the image plane so that it is orthogonal to the radiologist's viewpoint. 
The 3D scene can also be automatically annotated with labeled 3D axes to assist 
in the visual orientation by the radiologist. 

The volume-rendered image can be manipulated in various ways (i.e., 
using opacity maps, cutting planes, rotation, and fly-tliroughs). A second 
-14- 
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method for switching between the 2D and 3D viewers is to click on a 2D 
thumbnail image representation of a diagnostic finding (or its appended 
secondary 2D and 3D images) shown in an intermediate report display, thereby 
recalling the last state of the 2D or 3D viewer associated with the newly 
activated finding. 

When transitioning between 2D and 3D viewers, the last state of each 
viewer is stored. For example, the proper grayscales (or color scales) and 
opacity maps are applied according to the last recalled W/L or Op settings, 
respectively. Similarly, when jumping to a previous finding by clicking on its 
thumbnail image representation, the last W/L and/or Op settings for that finding 
are recalled depending on whether the thumbnail represents a 2D or 3D image. 
A previous button, Pr, allows the radiologist to toggle between the two most 
recent W/L settings or Op settings in the 2D and 3D viewers, respectively. 
Alternatively, the user can press on the LMB followed by a click of the RMB to 
activate the Pr function. 

Usually a radiologist compares a current examination to previous 
examinations by way of old reports and/or images. A conventional radiology 
report and images can be considered to represent unorganized sets or collections 
of diagnostic findings. Due to the database organization of diagnostic findings 
created by this invention, the system has the ability to aid in analysis of a current 
examination, by presenting, at step 311, the radiologist with an organized 
display of significant prior findings extracted from the previous reports retrieved 
by the file loader. 
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As shown in Fig. 9 A, the prior findings may be divided into "Significant 
Prior Findings" and "All Prior Findings", as shown in Fig. 9A. For example, the 
finding for the "Liver" shown in "All Prior Findings" section of the report does 
not appear in the "Significant Prior Findings" section of the report. This 
omission of the liver may be due to the fact that the "Liver" is unrelated to the 
"Heart," "Lung," or "Skeleton" of the 9/1/1 procedure or the CPT code for the 
9/1/1 procedure. Another reason for the omission may be that the "Liver" 
finding was not designated with a sufficient level of significance. The prior 
findings may be sorted by a designated priority code, anatomy, or modality (e.g., 
x-ray). In addition, the presentation of prior findings may include stored images 
associated with the findings, as shown in Figs. 9B and 9C. In the case where the 
prior findings are part of a horizontally linked series, described below, the 
radiologist may be presented with a trend report which shows a chronological 
tracking of a finding, such as tumor size, as shown in Fig. 10. 

During review of an image using the viewers as described above, the 
radiologist searches for any diagnostically significant image features. When the 
radiologist locates a diagnostically significant feature, the radiologist begins the 
process of recording a diagnostic finding at steps 106 and 3 12. The process of 
recording a diagnostic rinding begins with positioning the cursor over the 
location of the feature on the digital image and clicking the RMB at step 312. 
Alternatively, when applying the invention to conventional x-rays or images, a 
digital camera device can be pointed at an image finding, and a representative 
digital image can be recorded. Alternatively, the radiologist may point at the 
feature by using an intelligent view box. Clicking on the RMB stores the image 
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coordinates, for example DICOM coordinates, and an image number 
corresponding to the cursor location in a database. To complete the definition of 
a diagnostic finding, an anatomical-pathological code and, optionally, 
secondary attributes are assigned to the image coordinates and automatically 
stored in the database. The anatomical code identifies the anatomical location 
within the body, and the pathological code describes the radiologic pathology of 
the identified image feature. The anatomicahpathological codes maybe derived 
from a predefined lexicon, such as the American College of Radiology (ACR) 
Index of Radiological Diagnoses or Systematized Nomenclature of Medicine 
(SNOMED). The secondary attributes provide additional descriptions of the 
finding and include, for example distance, area and volume measurements, 
characteristics and status of the finding, as well as multimedia information such 
as audio descriptions, 3D snapshots, and 3D illustrated movies. 

In response to the RMB click the reporting system can automatically 
display a pop-up anatomical-location menu at step 314. The anatomical-location 
menu may consist of a cascading list of anatomical location codes that have been 
customized based on the previously verified CPT and ICD codes; i.e., the 
anatomical-location menu presents only the anatomical organs associated with a 
particular radiologic examination. The cascading anatomical-location menu 
provides greater levels of detail of the finding's anatomical location with each 
cascading level presented. For example, a first level might specify "Colon", and 
a second level "Sigmoid Colon". Upon selection of an anatomical code, the 
reporting system displays a cascading pathology-code menu, at step 316, which 
displays a cascading list of pathology codes that correspond to the selected 
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anatomical location. For example, a first level of the pathology-code menu 
might specify "Neoplasm", the second "Benign Neoplasm", and the third 
"Polyp". An anatomical:pathological code must be assigned to any unlabeled 
findings prior to final report approval; otherwise, these findings are labeled with 
the default "unknown locationrunknown pathology" or any combination thereof. 
If no abnormal findings are entered for a particular anatomy (i.e., for anatomical 
organs contained within the scope of an examination described by a CPT code), 
then a "Normal" default may be applied as a summary diagnosis for that organ. 
When a diagnostic finding has an indeterminate etiology, the radiologist my 
assign a list of diagnostic possibilities, representing a differential diagnosis, as 
secondary attributes of that finding. Alternately, the reporting system 10 can 
incorporate voice activated control and natural language processing in 
conjunction with or instead of the pop-up annotation menus, i.e. the anatomical- 
location and pathological-description menus. The radiologist could speak 
"Sigmoid Colon Polyp" to achieve the same result as using the annotation 
menus. 

As each diagnostic finding is created, a representative thumbnail image 
620, as shown in Fig. 6, may be displayed on the right side of the 2D and 3D 
viewers for immediate presentation and recall, and the thumbnail images later 
may be incorporated into the final report. Alternately, the report with thumbnail 
images can be displayed on a second independent monitor as it is being created. 
The above method for entering an anatomicakpathological code is denoted 
"click and label". Two alternative methods are also possible for performing 
steps 314 and 316. 
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The first alternative method, "click-pause-label", allows the radiologist 
to postpone assignment of an anatomicahpathological code until sometime later 
during the analysis of the finding, in this case, the radiologist must deliberately 
press anatomy-location and/or pathology-description button, An and Pa, on the 
2D or 3D viewer, as shown in Fig. 6, to subsequently activate the corresponding 
pull-down (as opposed to pop-up menu) annotation menu. The second 
alternative method, "click-click-click and label-label-label", allows the 
radiologist to annotate the diagnostic findings during final report editing. A 
more detailed description of these two methods is discussed below in 
conjunction with the method of operation of the reporting system. 

The method of entering and annotating diagnostic findings is not limited 
to computer pop-up and pull-down menus containing preselected terminology. 
Keyboard, voice recognition, macros, and natural language processing are 
available to enter diagnostic findings and secondary attributes. 

After assignment of the anatomicahpathological codes, secondary 
attributes may added at step 318 to embellish or support the diagnosis. As 
shown in Fig. 6, the user-interface 600 of the reporting system 10 includes 
various options for adding secondary attributes. A chart of the symbols used on 
Fig. 6 are set forth in the following chart. The options may be available in one 
or more of three major subsystems of the invention, as shown in the third 
column of the chart. The three major subsystems include image analysis (IA), 
report generation (RG), and report viewing (RV). The image analysis subsystem 
includes the image loading, step 100, and a display of the images, step 104. The 
report generation subsystem includes the recording of the findings, step 106, 
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annotation of findings, step 108, and creation of links/groups, step 110. The 
report review function includes algorithms associated with step 1 12. 





Annotation menu listing ACR 
Dx codes 


RG 


Vo 


Volume measurement button 


IA 


Ch 


Characteristic button 


RG 


Di 


Distance measurement button 


IA 


At 


Area measurement button 


IA 


Au 


Audio record button 


RG 


Pt 


Priority button 


RG 




Recommendation button 




Sn 


Snapshot button 


IA&RG 


Mo 


Movie button 
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W/L J. 


Window/Level presets menu 


IA 




Orientation button 


IA 
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Overview button 


IA 


Pr 


Previous window/level setting 
toggle button 




2D/3D 


2D/3D viewer toggle button 


IA 


Cr 


Cursor on/off toggle button 


IA 


Series 


Series menu 


IA 


MPR 


Multi-planar button 


IA 


Surf 


Surface rendering button 


IA 


Op J, 


Opacity map presets menu 


IA 


« * 


Render box size menu 


IA 




Opaque cutting plane toggle 
button 


IA 
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For example, a characteristics button, Ch, is included to activate a menu 
of descriptive attributes that enhance a specific diagnostic code set, (i.e., 
anatomy:pathology code combination). For example, "liverimetastatic neoplasm 
from colon" (ACR diagnostic code 761.3375) can be further characterized with 
the number of lesions (i.e., single or multiple). 

A distance-measurement button, Di, is included in the user-interface of 
the reporting system 10 to permit the radiologist to measure a finding in the 2D 
or 3D viewer with any number of diameters. Similarly, an area-measurement 
button, Ar, allows the radiologist to define a region-of-interest (ROI) from 
which the cross-sectional area, mean pixel or voxel value (depending on whether 
the measurement is made on a 2D or 3D image, respectively), and standard 
deviation of pixel or voxel values in that region can be calculated. 
Measurements automatically become secondary attributes of the active 
diagnostic finding and are stored in the database associated with the diagnostic 
finding. Additionally, a volume-measurement button, Vo, is provided to permit 
the radiologist to define a volume-of -interest VOL The reporting system 10 can 
create the VOI by 3D segmentation means, as disclosed in U.S. Patents 
5,782,762, 5,920,319, and 6,083,162, each of which are incorporated herein by 
reference. A volume measurement calculated from the VOI may be added as a 
secondary attribute. 

The reporting system also permits the assignment of both priority levels 
and recommendations to a finding. A priority button, Pt, permits the radiologist 
to add a certain level of significance to a diagnostic finding at step 319. For 
example, the priority level may be selected from a predefined list containing 
-21- 



WO 03/046810 PCT/US02/37214 
entries such as "(1) Life-Threatening", "(2) Significant", "(3) Index", "(4) 
Repetitive", and "(5) Incidental", and such entries may be presented as part of 
any subsequent report, as shown in Fig. 9B. One of more of priority levels may 
be combined, such as "Life-Tlireatening" and "Index". An "Index" priority 
denotes a trackable finding, as described below. The priority level can also be 
used to generate the production of the "Significant Prior Findings" and "All 
Prior Findings" as shown in Fig. 9A. For example, the "Significant Findings" 
may include prior findings designated with a priority of level of "Life- 
Threatening" or "Significant." Hence a selected priority level or set of levels 
can be used to select the findings for inclusion as "Significant Prior Findings" in 
the report of Fig. 9 A. A recommendation button,, Rm, can be used to label a 
"Sigmoid Colon: Polyp" diagnostic code with a recommendation such as 
"Recommend colonoscopy for polyp removal." By default, the reporting system 
10 does not assign any particular priority or recommendation to a diagnostic 
finding; however, certain diagnostic codes may be assigned default priority and 
recommendation codes which may be changed by the radiologist. 

An audio button, Au, is included in the user-interface to allow the 
radiologist to dictate a verbal description of a diagnostic finding, and that audio 
file becomes a secondary attribute of the finding. The audio file can be saved in 
the final report unchanged, or it can be transcribed to text by a typist or a voice 
recognition system. 

A snapshot button, Sn, in the user-interface allows the radiologist to 
record any number of additional 2D and 3D images as secondary attributes of a 
diagnostic finding. For example, a "colon:polyp" diagnostic finding could be 
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supported by additional 3D snapshots of the polyp. In the case of 
"spine:arthritis" which is seen over a large portion of the skeleton, a single 
diagnostic finding can be created to establish the diagnosis, and additional 
snapshots of other sites of the disease can support the diagnosis. Alternatively, 
creating multiple individual diagnostic findings documenting arthritis could 
achieve the same result. Additionally, the recording system provides the ability 
to place a marking symbol in the 2D or 3D images indicating the location of the 
selected feature. The snapshot function also records the location of the marking 
symbol visible within the 2D or 3D viewer, as well as the state of the 2D or 3D 
viewer at which time the Sn button was pressed. 

A movie button, Mo, functions in a similar manner by appending cine 
clips of moving 2D or 3D images, including active annotations and voice 
descriptions. The active annotations can take the form of freehand notations 
"drawn" over the 2D or 3D images during recording of the cine clip. The drawn 
freehand notations can be similar to "chalkboard-style" markings used by 
television commentators to diagram and analyze football plays. 

To assist radiologists in establishing a diagnosis, the annotation menus 
may also provide links to reference materials and example images related to 
each potential diagnostic code set combination. The annotation menus may 
include options to undo accidental RMB clicks. The reporting system 10 also 
permits the radiologist to recall the annotation menus to reassign a diagnostic 
code to a particular finding if the diagnosis is revised during the evaluation 
process. 
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The reporting system 10 may also perform computerized diagnoses at 
step 320. For example, computer-assisted polyp detection (CAPD), as disclosed 
in U.S. Patent 5,920,319, can be integrated with the system so that 
CAPD-identified polyps can be automatically correlated with radiologist-defined 
polyps by correlating the proximity (i.e., Euclidean distances) of image finding 
coordinates. The identified diagnostic findings can be used to support advanced 
applications, such as the creation of "polyp maps" for subsequent endoscopic or 
surgical guidance. A polyp map consists of a 3D-rendered colon with 
highlighted polyp locations. 

Another example of an advanced application that this reporting system 
supports is a Transbronchial Needle Aspiration (TBNA) targeting scheme. The 
TBNA application uses the stored data in the reporting system 10 to 
automatically construct airway models and lymph node targets (i.e., 
surface-rendered models of the anatomy generated using the respective finding 
coordinates). TBNA is a bronchoscopy technique that permits a needle biopsy 
of suspicious mediastinal lymph nodes. The tracheobronchial tree and lymph 
nodes are defined by their diagnostic finding coordinates, respectively, and are 
assigned secondary attributes by the radiologist to indicate the TBNA lymph 
nodes as targets. Further refinement of the lymph node targets (i.e., modeling 
lymph nodes as spherical or ellipsoid objects) can use the distance, area, and 
volume measurements that are generated as secondary attributes of those lymph 
nodes. 

After the review of the image(s) is deemed complete, the report display 
is presented for the radiologist to review at step 332. The report display is 
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invoked by pressing a report button in the user-interface to activate the report 
display. Alternately, when using a two-monitor system or a wide monitor 
display, the report can be shown simultaneously as it is being generated. The 
reporting system 10 sorts the diagnostic findings according to anatomical 
categories, with high priority findings placed at the top of each category. The 
reporting system 10 can also order the findings by priority levels, irrespective of 
anatomical categories, or by chronology, indicating the order in which the 
findings were recorded by the radiologist. The reporting system 10 highlights 
each high-priority finding (e.g., life-threatening and significant findings, levels 1 
& 2) with color-enhanced text. The radiologist edits the final report as 
necessary, including linking redundant findings at step 324. 

A powerful feature of the paradigm's report format and database 
structure is the ability to link and track diagnostic findings within the same 
examination (i.e., vertical linking) and across serial examinations (i.e., 
horizontal linking). For example, a CT examination generally consists of a 
hierarchy of series/acquisitions/images. A diagnostic finding identified on an 
image within one series may also be seen in another series of the same 
examination. The reporting system 10 provides the ability to vertically link 
(group or combine) such diagnostic findings within its database. In one 
implementation, the radiologist "drags and drops" a finding onto a matching 
finding in the report display to achieve linking, and the "dropped" finding 
becomes a subset of the primary finding. Alternatively, the reporting system 1 0 
could perform linking via a command-line interface, voice-activated control, or 
graphical interface, i.e., highlight user-selected related findings and press a 
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"Group" button. The purpose of vertical linking is to manage redundancy of 
report information. 

Similarly, the reporting system 10 provides horizontal linking as a means 
to track and monitor a diagnostic finding over time and across various imaging 
modalities, hi horizontal linking, diagnostic findings can be "dragged and 
dropped" across reports. In this case, the diagnostic findings exist independently 
in their respective reports and do not necessarily become subsets of other 
findings. Horizontal linking provides a means to efficiently analyze a particular 
diagnostic finding over time (i.e., disease tracking). As illustrated in the report 
of Fig. 10, the size of a tumor can be automatically presented in graphical format 
as size versus time. The size of the tumor can be selectively measured as an 
axial distance, area, volume, or other function. 

An extension of "linking" is "compositing." A group of image findings 
(e.g., pleura:pleural effusion, heart: cardiomegaly, lung:pulmonary edema) can be 
composited (i.e., linked or grouped) by the radiologist or by an artificial 
intelligence (AT) program to yield a cumulative diagnosis of "congestive heart 
failure." Similarly, the radiologist or an AI program can composite other 
clinical information (e.g., laboratory values or pathology reports) to support and 
establish a diagnosis. For example, specific findings may be grouped by use of a 
"Group" button followed by selection of specific annotations to describe the 
grouped set of findings. 

The reporting system 10 also allows for the automatic incorporation of 
repetitive findings (i.e., designated as a "Repetitive finding" using the above 
priority assignment feature) from previous reports into a new report (e.g., 
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evidence of prior gallbladder surgery). If a previous report contains a 
"trackable" finding (i.e., designated as an "Index finding" using the above 
priority assignment feature), that previous finding is brought to the attention of 
the radiologist. In this case, the trackable finding can be linked horizontally 
across reports, and the temporal progression of this finding, as shown in Fig. 10, 
can be observed in a specialized viewer. 

The report display also includes a suspend-resume button for suspending 
or resuming an examination in case the radiologist is interrupted during the 
review. Upon completion of the report, the reporting system 10 stores and sends 
the final report, as shown in Figs. 7A-C, at step 326. The reporting system 10 
may issue the report by any combination of telephone, fax, pager, e-mail, or 
wireless PDA and may include return receipt verification. The automated 
sending and receipt verification allows the radiologist to quickly communicate 
his or her findings and track this communication. Along with the prioritized and 
highlighted presentation of the most significant findings, the automated sending 
feature of the reporting system 10 helps to fulfill the radiologist's duty for timely 
communication of results and follow-up on the findings. 

The reporting system also supports "real-time dynamic radiology." Each 
diagnostic finding is annotated with a timestamp. After an initial report is 
"signed off," any future changes to the report can be recorded as a history of the 
report. Any subsequent significant changes can be automatically communicated 
to a clinician and verified upon their receipt. 

The reporting system 10 monitors how the radiologist reviews an 
examination. The final report can also indicate how much time a radiologist 
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spends reviewing an exam, number of findings, and average time per finding. 
Statistics, including total review time, time per finding, number of findings, and 
diagnostic accuracy, are compiled during a review session and are reported as 
needed. This feature creates a utilization management and quality assurance 
measure that is appealing to the Health Care Financing Administration (HCFA) 
and health maintenance organizations (HMOs). In addition, physician profiling 
may be employed for utilization management to review the ordering practices of 
referring physicians. 

The final report can also be automatically translated into a foreign 
language using the standardized lexicon of anatomical:pathological codes and 
simple lookup tables. 

Healthcare organizations further benefit from the automation and 
efficiency of the system. In particular, billing speed and accuracy are increased. 
Billing requires matching of ICD and CPT codes, a task that currently requires 
highly-trained personnel to decipher radiology reports and verify proper code 
assignments. Incorrect coding results in denied or delayed reimbursement by 
insurers. However, the present reporting system automates the process and 
allows radiologists to approve the coding process. 

The method of operation is best illustrated by its application in the field 
of radiology as shown in Fig. 4. Upon starting the software program, the 
radiologist signs in, with either a password or voice signature or any other 
security measure, to begin the evaluation at step 400. Secure sign-in protects 
access to the database and validates the identity of the radiologist generating the 
report. The file loader displays a work list of patients whose examination 
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studies are accessible. The radiologist selects the name of a patient at step 402, 
and the file loader displays all of the associated unread examination files. The 
radiologist selects a particular examination file, and that examination file is 
loaded into computer memory. 

The file loader displays the CPT and ICD codes assigned to a particular 
examination. This information can be obtained from the HIS 36 or entered 
manually. The radiologist verifies the CPT and ICD codes and makes any 
necessary changes at step 404. Correct assignment of the CPT and ICD codes 
by the radiologist is essential for electronic billing and expedited reimbursement 
by insurers. 

After validation of the CPT and ICD codes, the radiologist begins 
analysis of the first image presented in the 2D viewer or selects an alternate 
image, at step 406, from the series menu which lists all of the images or sets of 
images (i.e., series) in a patient exam available for review. The radiologist may 
change the displayed image in order to locate diagnostically significant features 
in other images at step 408. For example, the radiologist may press the 1MB 
while moving the mouse to pan through multiple images in the 2D viewer 
(provided that more than one image is contained in the series). The radiologist 
may also translate the displayed image up, down, and sideways by pressing the 
MMB while moving the mouse. The radiologist may also zoom the displayed 
image by pressing the 1MB and MMB simultaneously while moving the mouse. 
In the 3D viewer, the mouse operations are similar except that pressing the LMB 
while moving the mouse causes the 3D rendered scene to rotate in space or to 
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guide a "fly-through." Alternatively, multiple images or series can be displayed 
simultaneously in separate windows in the image analysis (IA) viewer. 

To aid in the identification of diagnostically significant features, the 
radiologist may toggle between 2D and 3D viewers by pressing the 2D/3D 
toggle button as shown in Fig. 6. When the 3D viewer is initially activated, a 
volume-rendered image centered around the coordinates of the identified feature 
is created (i.e., a cube of CT data is volume-rendered). The radiologist may 
adjust the size of the volume (i.e., cube of digital data) that is rendered in the 3D 
viewer via the render-box-size menu. The radiologist may further adjust the 
volume-rendered image in various ways, such as using opacity maps, cut planes, 
and rotation. MPR and surface rendering can also be activated in the 3D viewer. 

When the radiologist toggles between 2D and 3D viewers, the last state 
of each viewer is recalled. The radiologist may also toggle between the 3D and 
2D viewers by clicking on a primary 2D thumbnail image representation of a 
diagnostic finding (or its supporting secondary 2D and 3D thumbnails), thereby 
recalling the last state of the 2D or 3D viewer associated with the activated 
finding. The cursor position and location of any marking symbols in the display 
are recalled as part of the last state of the viewer. The 2D or 3D viewer then 
enters an edit mode, during which the radiologist can append additional 
secondary attributes to the activated diagnostic finding, and these are 
subsequently stored in proper locations within the database. 

The radiologist can also set the orientation of the image data prior to 
image analysis. If an image or image series needs to be reoriented, the 
radiologist pans through the volume of images to locate the most superior image 
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(or close to it). Then, the radiologist toggles the orientation button, at which 
time the viewer goes into an orientation mode. The radiologist rotates the image 
plane by pressing the LMB and moving the mouse until the proper 
anterior/posterior and left/right orientation is achieved. Finally, the radiologist 
toggles the orientation button again to set the proper orientation. The viewer 
automatically adjusts the 2D image plane so that it is orthogonal to the 
radiologist's viewpoint. 

The radiologist has further control over the display of the images such as 
W/L (i.e., grayscale or color scale) and 3D opacity maps settings. The 
radiologist may toggle between the two most recent W/L settings or Op settings 
in the 2D and 3D viewers by pressing the previous button, Pr, as shown in Fig. 
6, or simultaneously pressing the LMB and RMB. Additionally, the radiologist 
may toggle a visible cursor on and off by pressing a cursor-toggle button, Cr, as 
shown in Fig. 6, to indicate the location of a finding in both the 2D and 3D 
viewers. By pressing the overview button, the radiologist re-centers a 2D or 3D 
volume-rendered image in case the scene is moved out of sight. 

When the radiologist locates a diagnostically significant feature, the 
radiologist positions the cursor over the location of the feature on the digital 
image and clicks the RMB to mark the feature at step 410. Clicking on the 
RMB stores the image coordinates and image number corresponding to the 
cursor location in database. To complete the definition of a diagnostic finding, 
the radiologist annotates the point (location) by assigning an 
anatomicabpathological code and optionally assigning secondary attributes at 
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steps 412 and 414. This annotation is stored in the database, and it may also be 
displayed as a text overlay on the image. 

The radiologist selects an anatomical:pathological code from a 
predefined lexicon, such as the ACR Index of Radiological Diagnoses or 
SNOMED or a custom designed lexicon, to create a diagnostic finding. As each 
diagnostic finding is created, a representative thumbnail image 620 maybe 
displayed on the right side of the 2D and 3D viewers, or in a separate display, 
for immediate review and recall, and the thumbnail images later may be 
incorporated into the final report as shown in Figs 7B and 7C. 

The radiologist enters the anatomical:pathological code by one of several 
modes. In a first mode, "click and label", cascading pop-up annotation menus 
are presented to the radiologist immediately after a feature is marked by an RMB 
click at step 500 of Fig. 5 A. The radiologist selects an appropriate anatomical 
location description from the anatomical-location menu at step 502. For 
example, the radiologist may select Colon: Sigmoid colon. After the selection, 
the radiologist selects the pathological description from the pathology- 
description menu at step 502. For example, the radiologist may select 
Neoplasm:Benign Neoplasm:Polyp. A secondary attribute may then be assigned 
at step 504. 

La a second mode, "click-click-click and label-label-label", the 
radiologist identifies all the diagnostically significant features first and 
subsequently annotates the features with diagnostic codes and secondary 
attributes. As shown in Fig. 5B, the radiologist marks a designated feature at 
step 550 and then proceeds to mark successive features by repeating step 550. 
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After all desired features are marked, the radiologist assigns a diagnostic code to 
each marked feature by assigning an anatomical code at step 552 and a 
pathological code at step 554. Secondary attributes are assigned at step 556 
either following the marking of a feature at step 550 or the assigning of 
anatomical and pathological codes at steps 552 and 554. The radiologist must 
assign a diagnostic code to any unlabeled findings prior to final report approval; 
otherwise, these findings may be labeled with a default "Unknown 
Location:Unknown Pathology." Additionally, the radiologist may recall the 
annotation menus to reassign an anatomicakpathological code to a particular 
finding if the diagnosis needs to be revised during the evaluation process. 

The radiologist may also assign secondary attributes to embellish or 
support a diagnostic finding at step 414, but secondary attributes are not required 
for establishing a diagnostic finding. The radiologist may enter descriptive 
characteristics, dimensional measurements, audio descriptions, and specific 
snapshots of particular views of the identified finding as secondary attributes. 
For example, the radiologist may add descriptive characteristics that enhance a 
specific diagnostic code set from a characteristics menu of descriptive 
characteristics. 

The radiologist may measure one or more dimensions of a finding, for 
example, a diameter of an identified feature in the 2D or 3D image. The 
radiologist activates the distance measuring function by pressing the distance- 
measurement button, Di, as shown in Fig. 6. The radiologist measures the 
distance by clicking on first and second object points which span the 
characteristic length. Similarly, the radiologist may measure the area of an 
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identified feature by pressing the area-measurement button, Ar, as shown in Fig. 
6 and defining a region-of-interest (ROI) using the input device 27. The 
cross-sectional area, mean pixel or voxel value, and standard deviation of pixel 
or voxel values in the ROI can be calculated. The radiologist may also add a 
volume-measurement as a secondary attribute by pressing the volume- 
measurement button, Vo, as shown in Fig. 6. 

As part of step 414, the radiologist may also add a priority level and 
recommendation to the diagnostic finding by pressing the priority button, Pt, or 
recommendation button, Rm, respectively, as shown in Fig. 6. In addition, the 
radiologist may append a verbal description of the diagnostic finding in the form 
of an audio file. To add a verbal description the radiologist presses the audio 
button, Au, as shown in Fig. 6 to initiate recording and then dictates a verbal 
description of the diagnostic finding. The radiologist presses the audio button 
again to stop recording, and an audio file of the verbal description is stored in 
the database attached to the finding. Audio files can be attached to "grouped or 
linked" findings or attached to individual snapshot images or movies. Audio 
files maybe edited and/or appended with additional audio clips. 

Additionally, the radiologist may record snapshots of any of the 
displayed 2D and 3D images as a secondary attribute by pressing the snapshot 
button, Sn, as shown in Fig. 6. For example, the radiologist may record any 
number of additional images showing differing views of a particular 
diagnostically significant feature. For example, a "colon:polyp" diagnostic 
finding could be supported by additional 3D snapshots of the polyp. The 
radiologist may also append cine clips of moving 2D or 3D images (including 
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audio and active annotations) as a secondary attributes in a manner similar to 
recording snapshots by pressing the movie button, Mo, as shown in Fig. 6. 
Pressing of the movie button starts and stops the recording of the cine clip. 

Prior to final report review, the radiologist may also invoke computer- 
aided location and analysis of diagnostically significant features, at step 416, 
whereby the system automatically identifies and diagnoses suspicious image 
features. For example, the radiologist can review polyps found by the CAPD that 
were not previously identified by the radiologist. 

After the radiologist's review is deemed complete, the radiologist clicks a 
report button on the bottom of either the 2D or 3D viewer as shown in Figs. 6 to 
activate the report display at step 418. Alternately, the report can be generated 
and simultaneously displayed on a. second monitor while the diagnostically 
significant findings are being located and annotated. The diagnostic findings 
are sorted according to anatomical categories and priorities, with high priority 
findings being placed at the top of each category. Each high-priority finding is 
highlighted with color-enhanced text. The sorting and highlighting of the 
diagnostic findings alerts the end-user to the most significant diagnostic 
findings. Alternatively, the findings may be sorted by chronological order. 

The radiologist edits the final report as necessary, including linking 
redundant findings at step 420. This process could also be automated with 
artificial intelligence and detaining algorithms. The step of creating links, step 
420, may be performed before or after the step of reviewing the report, step 418, 
as depicted in Fig. 1, where the step of creating the links, step 110, occurs prior 
to the step of reviewing the report, step 1 12. In one implementation of vertical 
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linking, the radiologist "drags and drops" a finding onto a matching finding in 
the same report display, and the "dropped" finding becomes a subset of the 
primary finding. Alternatively, the radiologist can form links via a 
command-line interface or voice-activated commands (control). Similarly, the 
radiologist may assign horizontal linlcing to track and monitor a diagnostic 
finding over time and across various imaging modalities (i.e., disease tracking). 
In horizontal linking, diagnostic findings can be "dragged and dropped" across 
reports in a similar fashion. 

The radiologist may also composite a group of image findings to yield a 
diagnosis as illustrated above for "congestive heart failure." In this process, the 
radiologist or an AI program can linlc/group/composite additional clinical 
information (e.g., laboratory and pathology report values) to support a diagnosis. 

The radiologist further reviews any repetitive diagnostic findings from 
previous reports which are brought to the attention of the radiologist by the 
system. If a previous report contains a repetitive diagnostic finding (e.g., 
evidence of prior gallbladder surgery), that finding is presented to the radiologist 
for automatic incorporation into the new report. If a previous report contains a 
"trackable" diagnostic finding (e.g., index lymph node measurement), the 
radiologist can link the trackable diagnostic findings horizontally across reports, 
and the temporal progression of this diagnostic finding can be observed in a 
specialized viewer. 

The radiologist can suspend an examination for later resumption by 
pressing the suspend-resume button during the review. Upon completion of the 
report the radiologist instructs the system to send the report to the end-users 
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(e.g., clinicians) at step 422. Additionally, the end-user can access the report via 
a Web server after the report has been posted. As noted above, the report may 
be sent by a combination of telephone, fax, pager, e-mail, or wireless PDA and 
may include return receipt verification. The automated sending and receipt 
verification allows the radiologist to quickly communicate his or her findings 
and verify this communication. 

End-users receiving the radiologist's report can customize the display of 
the information to best suit their needs. For example, when the reporting system 
is integrated with a PACS and/or HIS system, the clinician can click on a 
thumbnail image in the final report to access the original PACS or HIS image 
data. For convenience the PAC or HIS image data may be displayed on one 
screen or display terminal while the report data is displayed on a separate screen 
or display tenninal. Additionally, the reporting system can automatically 
translate the radiologist's report into a different language for the end-user. The 
standardized lexicon of diagnostic findings supports rapid translation of reports 
to foreign languages by employing translation look-up tables. 

The reporting system of the present invention has further application 
beyond the preparation and delivery of reports. The ability of the reporting 
system to enter diagnostic findings into searchable databases readily supports 
data mining for clinical trials, epidemiology studies, and outcomes analyses. 

Additionally, the reporting paradigm supports radiologic training. For 
example, a radiology resident can issue a preliminary report indicating his or her 
findings, and the preliminary report can later be modified by an attending 
radiologist to indicate any corrections. In the latter case, the system 
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automatically informs the referring clinician of any significant changes. The 
history of report changes can be recorded with each finding (or changed finding) 
having a timestamp. The reporting scheme also supports standardized testing 
(e.g., replacement of the American Board of Radiology's Oral Board 
examination) by objectively measuring a student's performance. Such an 
objective performance measure could also assist in comparing a radiologist's 
performance to that of a non-radiologist. 

These and other advantages of the present invention will be apparent to 
those skilled in the art from the foregoing specification. Accordingly, it will be 
recognized by those skilled in the art that changes or modifications may be made 
to the above-described embodiments without departing from the broad inventive 
concepts of the invention. For example, while the above invention has been 
illustrated in terms of its application to the field of radiology, the invention is 
equally applicable to other fields of medicine as well as other image analysis 
fields such as satellite imagery and photography. It should therefore be 
understood that this invention is not limited to the particular embodiments 
described herein, but is intended to include all changes and modifications that 
are within the scope and spirit of the invention as set forth in the claims. 
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Claims 

What is claimed is: 

1 . A computer implemented method for the analysis of image data 
comprising the steps of: 

presenting an image of an object to a user for review; 

associating a position descriptor with a selected feature of the object, the 

position descriptor representing the position of the selected 

feature within the image; 
associating a location-code with the selected feature, the location-code 

representing the location of the selected feature relative to a 

structure depicted in the image; 
associating a physical property characteristic with the selected feature, 

the characteristic representing a physical property of the selected 

feature; and 

storing the position descriptor, location-code, and physical property 
characteristic in a computer database, whereby an image finding is created. 

2. A method according to claim 1, wherein the image comprises 
anatomical information, and the location-code comprises a location of the 
selected feature relative to an anatomical structure depicted in the image. 

3 . A method according to claim 1 , wherein the physical property 
includes at least one of a pathology and dimension of the selected feature. 

4. A computer implemented method for creating a report pertaining 
to the diagnostic analysis of image data comprising the steps of: 

displaying an image of an object from the image data to a user for 

review; 

creating in a computer memory a diagnostic finding comprising the steps 
of 

a) identifying a feature shown in the object, 
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b) recording in the computer memory an image-detail of the 
identified feature, 

c) associating a physical property characteristic with the 
diagnostic finding, the characteristic representing a physical 
property of the diagnostic finding; and 

generating a report comprising the diagnostic finding. 

5 . A method according to claim 4, wherein the image-detail 
comprises a snapshot of the feature. 

6. A method according to claim 4, wherein the physical property 
includes at least one of a pathology and dimension of the selected feature. 

7 . A computer implemented method for analysis of an image 
comprising the steps of: 

associating a position descriptor with a selected feature shown in the 

image, the position descriptor representing the position of the 

selected feature within the image; 
associating a physical property characteristic with the selected feature, 

the characteristic representing a physical property of the selected 

feature; and 

storing the position descriptor and the physical property characteristic in 
a database, 
whereby an image finding is created. 

8. A method according to claim 7, wherein the physical property 
characteristic comprises at least one of a location description which describes 
the location of the selected feature relative to an object depicted in the image 
and a pathology description of the selected feature. 
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9. A method according to any one of claims 1, 4, and 7, comprising 
retrieving a content-code associated with the image, the content-code specifying 
that a specific object is depicted within the image, and initializing a menu in 
response to the content-code, whereby menu items associated with the specific 
object are listed in a predetermined manner in the menu. 

10. A method according to claim 7, wherein the content-code 
comprises at least one of an anatomical-location code and a pathology- 
description code. 

11. A method according to any one of claims 1 , 4, and 7, comprising 
retrieving a prior report containing a most recent physical property 
characteristic, and initializing a menu in response to the most recent physical 
property characteristic, whereby menu items associated with the most recent 
physical property characteristic are listed in a predetermined manner in the 
menu. 

12. A method according to any one of claims 1, 4, and 7, comprising 
retrieving a prior report containing a plurality of physical property 
characteristics, and initializing a menu in response to the most frequently used 
physical property characteristic, whereby menu items associated with the most 
frequently used physical property characteristic are listed in a predetermined 
manner in the menu. 

13. A method according to any one of claims 1 , 4, and 7, comprising 
retrieving a selected patient characteristic associated with the image, and 
initializing a menu in response to the patient characteristic, whereby menu items 
associated with the patient characteristic are listed in a predetermined manner in 
the menu. 
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14. A method according to claim 13, wherein the patient 
characteristic comprises the gender of the patient. 

15. A method according to claim 13, wherein the menu comprises at 
least one of an anatomical-location menu and a pathology description menu. 

16. A method according to claim 15 comprising retrieving a prior 
report containing a most recent physical property characteristic, and initializing 
the menu in response to the most recent physical property characteristic, 
whereby menu items associated with the most recent physical property 
characteristic are listed in a predetermined manner in the menu. 

17. A method according to claim 15, comprising retrieving a prior 
report containing a plurality of physical property characteristics, and the a menu 
in response to the most frequently used physical property characteristic, whereby 
menu items associated with the most frequently used physical property 
characteristic are listed in a predetermined manner in the menu. 

18. A method according to any one of claims 1, 4, and 7, comprising 
retrieving a selected patient characteristic associated with the image, and 
initializing a menu in response to the patient characteristic, whereby menu items 
unrelated to the patient characteristic are not presented in the menu. 

19. A method according to claim 1 8, wherein the patient 
characteristic comprises the gender of the patient. 

20. A method according to any one of claims I, 4, and 7, comprising 
creating a summary of prior findings. 
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21. A method according to claim 20, wherein the step of creating a 
summary comprises retrieving a prior report relating to a patient shown in the 
image and extracting a prior finding from the prior report. 

22. A method according to claim 20, wherein the step of arranging 
the prior findings comprises omitting a selected prior finding from the summary. 

23 . A method according to claim 20, wherein the step of creating a 
summary comprises arranging the prior findings according to a selected 
significance of the prior findings. 

24. A method according to claim 23, wherein the step of arranging 
the prior findings comprises at least one of grouping the prior findings within 
the summary and ranking the prior findings within the summary. 

25. A method according to claim 23, wherein the significance of the 
prior findings includes at least one of a designated priority code, a designated 
anatomical feature, and an image modality. 

26. A method according to claim 25, wherein the priority code is 
selected from the group consisting of "life-threatening," "significant," and 
"incidental." 

27. A method according to claim 25, wherein the anatomical feature 
comprises a particular body structure. 

28. A method according to claim 25, wherein the image modality 
includes at least one of an x-ray, a sonogram, and a CAT scan. 

29. A method according to any one of claims 1, 4, and 7, comprising 
retrieving a plurality of prior findings of the selected feature from at least one 



-43- 



WO 03/046810 



PCT/US02/37214 



prior analysis of an image, and displaying an element of the prior findings in a 
graphical display to describe a chronology of the element. 

30. A method according to claim 29, wherein the element comprises 
the size of the selected feature. 

31. A method according to claim 30, wherein the size comprises at 
least one of an area, volume, and a distance of the selected feature. 

32. A system for the analysis of image data comprising: 

a display for presenting an image of an object to a user for review; 

a position annotator for associating with a selected feature shown in the 

image a position descriptor representing the position of the 

selected feature within the image; 
a location annotator for associating with the feature a location-code 

denoting the location of the feature relative to a structure depicted 

in the image; 

a physical property annotator for associating with the selected feature a 

physical property characteristic representing a physical property 

of the selected feature; and 
a memory for storing the position descriptor, location-code, and physical 

property characteristic in a computer database to create an image 

finding. 

33. A system according to claim 32, wherein the image comprises 
anatomical information, and the location-code comprises a location of the 
selected feature relative to an anatomical structure depicted in the image. 

34. A system according to claim 32, wherein the physical property 
includes at least one of a pathology and dimension of the selected feature. 
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35. A computer system for analysis of an image comprising: 

a position annotator for associating with a selected feature shown in the 

image a position descriptor representing the position of the 

selected feature within the image; 
a physical property annotator for associating with the selected feature a 

physical property characteristic representing a physical property 

of the selected feature; and 
a memory for storing the position descriptor and the physical property 

characteristic in a database to create an image finding. 

36. A system according to claim 35, wherein the physical property 
characteristic comprises at least one of a location description which describes 
the location of the selected feature relative to an object depicted in the image 
and a pathology description of the selected feature. 

37. A system according to claim 32 or claim 35, wherein the image 
comprises the content-code denoting that a specific object is depicted within the 
image, and the system comprises a menu initializer for initializing a menu in 
response to the content-code, whereby menu items associated with the specific 
object are listed in a predetermined manner in the menu. 

38. A system according to claim 37, wherein the content-code 
comprises at least one of an anatomical-location code and a pathology- 
description code. 

39. A system according to claim 32 or claim 35, comprising a report 
loader for retrieving a prior report containing a most recent physical property 
characteristic and comprising a menu initializer for initializing a menu in 
response to the most recent physical property characteristic, whereby menu 
items associated with the most recent physical property characteristic are listed 
in a predetermined manner in the menu. 
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40. A system according to claim 32 or claim 35, comprising a report 
loader for retrieving a prior report containing a plurality of physical property 
characteristics and comprising a menu initializer for initializing a menu in 
response to a most frequently used physical property characteristic, whereby 
menu items associated with the most frequently used physical property 
characteristic are listed in a predetermined manner in the menu. 

41 . A system according to claim 32 or claim 35, comprising a menu 
initializer for retrieving a selected patient characteristic associated with the 
image and for initializing a menu in response to the patient characteristic, 
whereby menu items associated with the patient characteristic are listed in a 
predetermined manner in the menu. 

42. A system according to claim 41, wherein the patient 
characteristic comprises the gender of the patient. 

43 . A system according to claim 4 1 , wherein the menu comprises at 
least one of an anatomical-location menu and a pathology description menu. 

44. A system according to claim 32 or claim 35, comprising a menu 
initializer for retrieving a selected patient characteristic associated with the 
image and for initializing a menu in response to the patient characteristic, 
whereby menu items unrelated to the patient characteristic are not presented in 
the menu. 

45. A system according to claim 44, wherein the patient 
characteristic comprises the gender of the patient. 

46. A system according to claim 32 or claim 35, comprising a finding 
analyzer for creating a summary of prior findings. 
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47. A system according to claim 46, wherein the rinding analyzer 
comprises a file loader for retrieving a prior report relating to a patient shown in 
the image and for extracting a prior finding from the prior report. 

48. A system according to claim 46, wherein the finding analyzer 
comprises a sorter for arranging the prior findings according to a selected 
significance of the prior findings. 

49. A system according to claim 48, wherein the significance of the 
prior findings includes at least one of a designated priority code, a designated 
anatomical feature, and an image modality. 

50. A system according to claim 49, wherein the priority code is 
selected from the group consisting of "life-threatening", "significant", and 
"incidental." 

51. A system according to claim 49, wherein the anatomical feature 
comprises a particular body structure. 

52. A system according to claim 49, wherein the image modality 
includes at least one of an x-ray, a sonogram, and a CAT scan. 

53 . A system according to claim 48, wherein the finding analyzer 
comprises a finding grouper for grouping the prior findings within the summary 
according to the selected significance. 

54. A system according to claim 32 or claim 35, comprising a graph 
generator for retrieving a plurality of prior findings of the selected feature from 
at least one prior analysis of an image and for displaying an element of the prior 
findings in a graphical display to describe a chronology of the element. 
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55. A system according to claim 54, wherein the element comprises 
the size of the selected feature. 

56. A system according to claim 55, wherein the size comprises at 
least one of an area, volume, and a distance of the selected feature. 

57. A computer implemented method for the analysis of image data 
comprising the steps of: 

presenting an image to a user for review; 

associating a position descriptor with a selected feature of the object, the 

position descriptor representing the coordinate of the selected 

feature within the image; 
associating a location-code with the selected feature, the location-code 

representing the location of the selected feature relative to an 

entity depicted in the image; 
associating a property characteristic with the selected feature, the 

characteristic representing a property of the selected feature; and 
storing the position descriptor, location-code, and property characteristic 
in a computer database, whereby an image finding is created. 

58. A method according to claim 57, wherein the image comprises 
anatomical information, and the location-code comprises a location of the 
selected feature relative to an anatomical structure depicted in the image. 

59. A method according to claim 57, wherein the property includes 
at least one of a pathology and metric of the selected feature. 

60. A computer implemented method for creating a report pertaining 
to the analysis of image data comprising the steps of: 

displaying an image of a scene from the image data to a user for review; 
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creating in a computer memory a finding comprising the steps of 

a) identifying a feature shown in the image, 

b) recording in the computer memory an image-detail of the 
identified feature, 

c) associating a physical property characteristic with the finding, 
the characteristic representing a physical property of the finding; 
and 

generating a report comprising the diagnostic finding. 
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New Paradigm Report 

Wake Forest University/Baptist Medical Center 

Patient Name.- DOE, JOHN Radiologist: VINING, DAVID 

Patient ID: I2345678 Referring Physician: WELBY, MARCUS 
Requisition** 8765432 1 

Sex *. Male Date of Exam: 10/19/1998 

DOB: 1/1/1942 Date of Report: 11/3/1998 



Indication: Colon cancer screening (V76.49) 

Patient Prep: Liquid diet, split-dose Fleets Phospho-soda,6astroview,C02 insufflation 

Technique: Virtual colonoscopy protocol-CT abdomen(CPT 74150), 
CT pelvis (CPT 72192), 3D reconstructlon(CPT76375) 

Findings 

Lungs: Normal. 
Liver: Normal. 
Spleen: Normal. 
Pancreas: Normal. 
Gallbladder: 

Findingl: Gallbladder: calculi (ACR 762.81) 

Exam:999, Serles:5, Acquisition: l,X,Y,Z-82.3f 36.3,-76.9 
Features: Status: Recommend clinical correlation 

Characteristics.- multiple gallstones 

Size-. 8 mm 

Voice; [click for voice description] 

Images: 



FIG. 7A 



WO 03/046810 



9/15 



PCTYUS02/37214 



Images: 

Adrenals: 
Kidneys: 
Colon: 

Finding la: 

Features: 
Images: 

Finding lb: 
Features: 

Images: 




Normal. 
Normal. 



Colon, rectum:adenomatous polyp(ACR 757.31 11) 
Exam: 999,Series: 5, Acquisition:l,X,Y,Z: 26.8,192,-3679 
Status:HIGH Priority- Requires surgical attention 
Characteristics: polypoid, located on left wall, surrounded by fluid 
Size: 10mm 

Voice: [click fo r voice description] 




Colon, rectum: adenomatous polyp (ACR 757. 3111 ) 
Exam:999,Series: 5,Acquisition:2,X,Y,Z: 3371,354.9,-161.3 
Status: HIGH Priority- Requires surgical attention 
Characteristics: polypoid, located on left wall 
Size: 10mm 

Voice: [click for voice description] 




FIG. 7B 
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Images: 




Skeleton: 

Finding l: Lumbosacral spine: degenerative joint disorder (ACR 33.77) 
Exam: 999,Series:5, Acqulsltlpn:1 l X,Y,Z:-27.7 f 38.2,-272,9 

Features: Status: n/a 

Characteristics: minimal hypertrophic changes 
Size: n/a 

Voice: [click for voice description] 



f v':..-.^«£ w j;:2 



Misc: 



Finding 1: Abdominal aorta: atherosclerosis (ACR 981.721) 

Exam: 999, Series: 5, Acquisition: l.X.Y.Z: 9.1,-25.0,-1779 

Features: Status: n/a 

Characteristics: minimal atherosclerosis 
Size: n/a 

Voice:[click for voice description] 

Images: 
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1/1/01 
CXR 

Fx: 

Heart: Cardiomcgaly 

Artificial aortic valve 
Lungs: Normal 
Skeleton-chest: Degenerative 



3/1/01 
CT Chest/Abd/Pelvis 

Fx: 

Heart: Cardiomegaly 
Lungs: Normal 
Skeleton-chest: Normal 
Liver. Hepatomegaly 
Spleen: Normal 
Pancreas: Normal 



671/01 
US Chest 



Fx: 

Heart ? 
Lungs: ? 
Skeleton: ? 

jx] Prior Findings 



Significant Prior Findings: 



Cardiomegaly, CT, 3/1/01; Artificial aortic valve, CXR, 1/1/01 
Norma l 



Heart: 

fcUHgSK- 

Skcleton-chcst Degenerative , CXR, 1/1/01 
Misc (chest): Pleura: eflusion, US, 6/1/01 

All Prior Findings: 

Heart Cardiomcgaly, CT, 3/1/01; Cardiomegaly, CXR, 1/1/0 1 ; 

Artificial aortic valve, CXR, 1/1/01 

■ Lungo: Normal 

Skeleton-chest: Degenerative, CXR, 1/1/0 1 
Mkc-chcst Pleura ^Effusion, US, 6/1/0 1 
Liven Hepatomegaly, CT, 3/1/0.1 

Splseee Norm al 



Panoroaa; 



■ Normal 



Misc-abdomen: Ascites, CT, 3/1/01 



Fig. 9 A 
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Fig. 9B 
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Fig. 9C 
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Sigmoid colon: Poiyp 
Size: 33.14 mm 
x 17.96 mm 



Fig. 10 
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